1. Field of the Invention
The present invention generally relates to systems for using biometric measurements for identification, and more particularly to use of imaging techniques for such identification.
2. Background Description
Visual biometric recognition systems have inherent accuracy limitations. The quality of the visual image is strongly affected by ambient lighting conditions and resulting artifacts such as glare, shadows, dimness, and loss of detail. In total darkness, no image is obtained. In dim light or when an IR flash illuminator is used, the image resulting may be flat and devoid of most details. Under any lighting condition, especially in dim light, visual face recognition is less accurate for dark skinned persons, resulting in a racial bias to systems which use only visual images for face identification. Under any lighting conditions, visual face imagery often can not discriminate between identical twins, detect the use of surgical or worn disguises, or distinguish a live person from a photograph or mannequin. When used in automated surveillance systems, visual imagery makes it difficult and computationally intense to locate and separate faces within a frame of imagery. When a surveillance or other system is to be activated based upon the detected presence of any person, generally another sensor such as proximity or motion alarm must be used due to the time required to determine human presence based upon detecting faces in visual imagery. When gait, or other body movements are to be recognized, use of visual imagery alone makes the tasks of isolating a person's image, segmenting his body parts, and tracking movements very difficult in complex backgrounds. Visual ID of fingerprints may not insure that the finger is present and that the subject is alive at the time of identification. Visual images of actual iris patterns may be confused by patterns imprinted on contact lenses.
Automated face recognition systems based upon visual imaging often use the manually or automatically designated location of the eyes to position, scale, rotate, and de-skew the image. When the person is wearing sunglasses, various approximations can be made as to where the eye are believed to be located. However, errors in those approximations produce significant errors in identification against large databases. Additional features extracted from visual imagery for use in recognition or identification often include metrics involving the relative distances between facial features which can be seen in the image. There are on the order of 14 such features which may be seen, including: inner and outer eye corners, nostrils, bow of the lips, edges of the nose base, outer edges of the mouth, head outline, eyebrows. Also, there are other features which are more subjectively located such as the highest point of cheekbones. In dim light, especially for dark-skinned persons or when sunglasses are worn, few of these features may be apparent. In addition, use of makeup or other disguises, or changes in facial expression may distort the position of the features and result in recognition errors. In the case of visual recognition systems, then, analysis of face metrics is recommended only for classification and not for positive identification except in cases of very small databases.
Infrared recognition systems also have inherent limitations. IR provides no color information, meaning that eye, skin, hair, clothing, and vehicle colors are not detected. While this makes IR ID not racially biased, and less vulnerable to being fooled by those attributes which can easily be disguised, the lack of color information is a serious limitation when it is necessary to provide information to a human response force which is to locate or apprehend a person based upon imagery collected. A second limitation to IR imaging is the fact that thermal infrared emissions do not penetrate many glasses and plastics. Therefore, it is often not possible to image a person through a vehicle window or building window, or to see his eye area behind eyeglasses. The effect of eyeglasses on a thermal infrared image is akin to the effect of sunglasses on a visual image. Thermal infrared images are essentially immune from variations due to lighting changes. Certain disguise methods, such as colorizing agents for skin, hair, or eyes, do not change the IR image. Other disguise methods such as altering eyebrows or the apparent size and shape of the mouth or the definition of cheekbones through use of cosmetics also do not change the IR image. Disguises which involve appliances inside the mouth may distort the IR image more than naturally-performed facial expression and speech-related changes, but do not introduce additional IR features. Other disguises which involve appliques to the skin surface (such as moles, scars, artificial facial hair, and nose prostheses) can block or partially block the IR emissions; they are readily apparent in the IR image due to anomalous spectral histograms of the local area of application.
Under any lighting condition, including total darkness, IR recognition is equally accurate for light and dark skinned persons, resulting in no racial bias to systems which use only IR images for identification. IR imagery can discriminate between identical twins, detect the use of surgical or worn disguises, and distinguish a live person from a photograph or mannequin. When used in automated surveillance systems, IR imagery makes it easy and computationally simple to locate and separate faces within a frame of imagery. When a surveillance or other system is to be activated based upon the detected presence of any person, the use of IR camera makes it fast and accurate to determine human presence based upon detecting faces in imagery. When gait, or other body movements are to be recognized, use of IR imagery makes the tasks of isolating a person's image, segmenting his body parts, and tracking movements very simple even in complex backgrounds.
Automated face recognition systems based upon thermal infrared imaging often use the manually or automatically designated location of the eyes to scale, rotate, and de-skew the image. When the person is wearing eyeglasses, various approximations can be made as to where the eye are believed to be located. However, errors in those approximations produce significant errors in identification against large databases. Additional features extracted from thermal infrared imagery for use in recognition or identification often include metrics involving the relative distances between facial features which can be seen in the thermal infrared image. There include features which may also be seen in visual images, such as: inner and outer eye corners, nostrils, bow of the lips, edges of the nose base, outer edges of the mouth, head outline, eyebrows. However, IR features also include anatomically specific elements such as the paths of blood vessels under the skin, branch points or apparent end points of blood vessels, localized hot spots associated with cuts, tumors, infection, lymph nodes, localized cold spots associated with moles, scars, broken noses. In any light, even for dark-skinned persons, on the order of 300 such features may be apparent. Due to the large number of IR features, there is sufficient data to perform reliable identification or recognition based upon IR feature metrics, if a sufficient portion of the face is available in the image, and if a suitable model is used to account for facial expression, fatigue and stress, and speech-related changes. Even better accuracy may be obtained by using IR image metrics for classification of an image, and then pattern recognition based on thermal contours or blood vessel patterns or minutiae matching for the final identification decision, especially in cases of very large databases.
When face recognition is used in cooperative thermal infrared identification systems, the system accuracy is improved and the computational complexity of the system is reduced if the subject's face is repositioned the same as during the enrollment process. This can be accomplished by having the subject stand in the same place relative to the camera, with the same body and head tilt and the same facial expression. Since persons generally cannot remember and repeat a pose to the extent desired, the system can provide feedback to the subject by displaying the enrolled image overlaid with the current image. The system can furthermore cue the subject by visual or audio clues to: stand closer or straighten your head. Subjects can learn to adjust their face position to get sufficient alignment. Since viewing their own visual facial image is more natural than viewing their IR image, the feedback images, both for the enrolled and current images, are shown using the visual images. The enrolled image is shown with reduced contrast and perhaps only a skeletonized version of the image including the head and major feature outlines. The current image is shown in greater detail, perhaps like a reversed mirror, overlaid on the enrolled image. This feedback use may be the only use of the visual camera in a cooperative system, or in addition the visual camera may be used for locating eyes behind glasses, providing color information, or other uses as described, as well as in additional uses implied by this document.
Covert identification of persons is often an important element of securing high risk targets. Surveilled persons are less likely to adopt measures to avoid identification (such as disguise and staying out of the camera's field of view) if the area has low or no apparent illumination. Many current visual surveillance systems employ infrared illuminators and visual cameras which are sensitive to “near infrared”, which is the spectral band just beyond visual, considered from 0.7 to 1.3 micron. A typical visual image using an IR illuminator is shown in 93 of FIG. 9. While the location of the eyes can be readily determined, many of the details seen in the visual image taken under daylight conditions of 91 are lost. However, when compared to a visual image taken under dim light conditions of 92, the IR illuminated image shown in 93 offers significantly more useful details for identification, although the color information is sacrificed. A thermal infrared image of the same person, as shown in 94, which would be the same in daylight or complete darkness, provides anatomical details missing from the other three images. While a simple disguise, manikin, or photograph would fool the visual systems producing images shown in 91, 92, and 93, the image of 94 produced by the thermal infrared imager demonstrates that a live person was present at the time the image was produced, and contains sufficient subsurface anatomical details to uniquely identify a particular person from a collection of people who all look, or are disguised to look, similar to the eye and therefore to the visual camera.
Comparing a thermal infrared image to a visual image can be done most directly when the eyes can be seen in both bands. The eyes can then be used for registration, alignment and scaling. If the subject is wearing glasses, other features must be used which may have greater variation between corresponding points seen in the two images. For example, the nose may produce shadows in the visual image which cause imprecision in designation of the outer corners of the nose tip, which may be precisely located in the thermal infrared image. Use of an integrated dual-band imager, which automatically aligns the thermal infrared and visual images, simplifies the implementation of the methods of this invention, but is not required.
3. Comparison to Prior Art
Numerous approaches to recognition using visual imaging of faces Have been considered. Most commonly they involve collections of face metrics from measurements of the relative distances between visual features such as the nostrils, sides of the nose, corners of the eyes, corners of the mouth, ends of the eyebrows, etc.
U.S. Pat. No. 4,975,969 to Tal discloses a method and apparatus for uniquely identifying individuals by measurement of particular physical characteristics viewable by the naked eye or by imaging in the visible spectrum. This reference defined facial parameters which are the distances between identifiable parameters on the human face, and/or ratios of the facial parameters, and teaches that they can be used to identify an individual since the set of parameters for each individual is unique.
Tal's approach utilizes visible features on the face, and therefore cannot be relied upon to distinguish between faces having similar visual features, for example as would be the case with identical twins. In addition, the “rubber sheeting” effect caused by changes in facial expression, the aging effects which cause lengthening of the nose, thinning of the lips, wrinkles, and deepening of the creases on the sides of the nose, all cause changes in the parameters and ratios of any particular person's face may be measured by anyone taking a photograph, and thereby used to select or disguise another person to appear to be that person. Therefore, the security provided by such a technique may not be adequate for unattended or highly sensitive locations.
Visible metrics in general typically require ground truth distance measurements unless they rely strictly upon ratios of measurements. Thus, such systems can be fooled by intentional disguises, and they are subject to variations caused by facial expressions, makeup, sunburns, shadows and similar unintentional disguises. Detecting the wearing of disguises and distinguishing between identical twins may be done from visible imagery if sufficient resolution and controlled lighting is available. However, that significantly increases the computational complexity of the identification task, and makes the recognition accuracy vulnerable to unintentional normal variations.
The second most common approach to face recognition utilizes a principal components analysis (eigenanalysis) of visual face images to develop a set of characteristic features for matching an unknown visual image against a database of known visual images. Faces are then described in terms of weighting on those features. The approach claims to accommodate head position changes and the wearing of glasses, as well as changes in facial expression. However, pre-processing for registration is essential to eigenvector recognition systems. The processing required to establish the eigenvector set may be extensive for large databases. Addition of new faces to the database requires the re-training of the system. Turk and Pentland's U.S. Pat. Nos. 5,164,992 and RE36,041 apply the eigenface approach to audience recognition. They reference U.S. Pat. No. 4,858,000 of Daozehng Lu, who uses infrared detectors to locate members of the viewing audience, who are then imaged with visual cameras.
The two primary face recognition methods of metrics and eigenfaces have been commercialized by Visionics and Visage, as well as many other companies worldwide. In every case, the systems use visual imagery, sometimes with the use of infrared detectors to locate faces in cluttered scenes, and sometimes with the use of infrared illuminators to allow the face to be seen under dim conditions. Accuracies vary between 55% and 90% depending on lighting conditions, the time between enrollment and subsequent identification, and other variables including skin color of the subject. Prokoski has issued patents relating to the use of both approaches to thermal infrared images. Neither matching method has yet been commercially applied to the use of thermal infrared imagery Biometric identification systems which use passively collected images from infrared cameras operating in the 2-15 micron spectral band are not yet available. The cost of those cameras, and the inability of infrared imagers to see through eyeglasses have been the major hindrances to commercialization of infrared identification. The current invention is aimed at removing those hindrances.
Eigenanalysis of thermal infrared images for identification is described in the Prokoski et al U.S. Pat. No. 5,163,094 which discloses defining “elemental shapes” in the surface thermal image produced by the underlying vascular structure of blood vessels beneath the skin. Depending on the environment of use, thermal facial identification may provide greater security over identification from visual images and may therefore be considered preferable. It is extremely difficult, if not impossible, to counterfeit or forge one face to look like another in infrared, whereas it is often possible to disguise one person to look like another in visible light. However, the use of elemental shapes is found in practice to be vulnerable to such variables as head rotation and tilt, ambient and physiological temperature changes, variations in imaging and processing systems, and distortions or obstructions in a facial image (e.g., due to eyeglasses).
In spite of those limitations, eigenanalysis of the elemental shapes of a thermal facial image has successfully been used for recognition. In a 1993 project for Hanscom Air Force Base, referenced in U.S. Pat. No. 6,173,068 several sets of elemental shapes were produced for each infrared facial image by imposing different thermal banding constraints. The totality of shapes are then analyzed with respect to a library of facial thermal images. Eigenshape analysis is used to compare the characteristics of shapes in each person's images. Eleven characteristics of each shape are considered, including: perimeter, area, centroid x and y locations, minimum and maximum chord length through the centroid, standard deviation of that length, minimum and maximum chord length between perimeter points, standard deviation of that length, and area/perimeter. Each person's image is then characterized by a set of 11-coefficient vectors. The difference in eigenspace between any two images is calculated to yield a measurement to which a threshold was applied to make a “match/no match” decision. Accuracy of 92% or better was achieved in non-cooperative faces-in-the-crowd applications.
Other issued and pending patents of Prokoski address the use of thermal infrared images for face and body part identification through analysis of thermal contours, thermal minutiae, and patterns of vascular segments. U.S. Pat. No. 6,173,068 extracts minutiae from passive thermal infrared images and uses the collection of minutiae similarly to the use of fingerprint minutiae for identification; in this case of the subject's face. Also taught in a pending patent of the inventor, thermal infrared images taken in surveillance applications can be correlated against visual images in a Watch List database to identify persons for whom only a visual reference image is available.